# 01 — Architecture & Data Sources

## Data flow per dashboard run
```
┌──────────────────────────────────────────────────┐
│ DataBento Historical API                          │
│  • GLBX.MDP3 → CME futures options (ES/NQ/GC)    │
│  • OPRA.PILLAR → equity-option chains (RUT/SPX)  │
│  • schemas: definition (strikes/expiries)        │
│             statistics (settlement OI + price)   │
└─────────────┬────────────────────────────────────┘
              │ raw_symbol + strike + expiration + OI
              ▼
┌──────────────────────────────────────────────────┐
│  databento_chain.py                              │
│   parse → normalized contracts list              │
│   contracts = [{exp, dte, strike, side,         │
│                 oi, mid, multiplier}, ...]      │
└─────────────┬────────────────────────────────────┘
              │ + Schwab spot via get_spot_via_schwab()
              ▼
┌──────────────────────────────────────────────────┐
│  options_metrics.py                              │
│   fill_greeks() → BS gamma/delta/vanna from mid  │
│   gex_per_strike() → signed GEX, call_gex,       │
│      put_gex, net_delta, cumulative_gex, flip    │
│   walls() / max_pain() / vanna_per_strike()      │
│   atm_iv() → OI-weighted ATM IV                  │
└─────────────┬────────────────────────────────────┘
              ▼
┌──────────────────────────────────────────────────┐
│  dashboard_visual.py                             │
│   compute_extended() per (symbol, DTE-bucket)    │
│   render HTML with embedded JSON + Chart.js      │
└──────────────────────────────────────────────────┘
              ▼
       reports/dashboard_visual_*.html
```

## Why two data sources

**DataBento gives us the chain + OI** but its parent-symbol queries don't include the underlying futures/index settlement price in the same pull. Pulling a second DataBento query just for spot would cost extra OPRA bytes for no good reason.

**Schwab gives us the spot for free** via `/marketdata/v1/quotes`:
- `/ES`, `/NQ`, `/RTY`, `/GC` resolve to the front-month contract automatically
- `$SPX`, `$NDX`, `$RUT` work as cash-index symbols
- ETFs (SPY, QQQ, IWM, GLD) also work

So Schwab plays a tiny but critical role: **spot anchor**. The chain data is all DataBento.

## Why this beats the original "Schwab-only" plan

The original 2026-05-11 plan (see [[Schwab Options Data API]]) was to use Schwab for everything. We discovered:

| Issue with Schwab | Fix with DataBento |
|---|---|
| `$SPX` returns zero OI (OPRA index entitlement gap on non-pro tier) | OPRA.PILLAR returns real SPX OI |
| No options on futures (`/chains` 400 on `/ES`, `/NQ`, etc.) | GLBX.MDP3 has CME futures options |
| Greeks return `-999.0` outside RTH | We compute Black-Scholes greeks ourselves anyway |
| Day-of-week sensitive (closes on Memorial Day → -999s) | DataBento historical is calendar-independent |

The Schwab API is still useful for the spot-price layer and was a great learning step.

## Filesystem layout

```
/home/dlitt/projects-personal/schwab-options/    ← LIVE (WSL ext4, fast)
├── schwab_auth.py            OAuth + token refresh for Schwab spot lookups
├── databento_chain.py        DataBento chain fetcher (+ multi-parent merge)
├── options_metrics.py        Black-Scholes greeks + per-strike aggregations
├── dashboard.py              "Classic" ETF-proxy dashboard (Schwab REST chains)
├── dashboard_db.py           Real-data dashboard (DataBento + Schwab spot)
├── dashboard_visual.py       Visual two-panel + bar-chart dashboard (this is the one)
├── reports/                  generated *.html + *.json per run
├── examples/                 reference screenshots (SpotGamma, Quant Data dashboards)
├── requirements.txt          httpx>=0.27, databento>=0.78
└── README.md

/mnt/c/Users/dlitt/OneDrive/claude/projects-personal/schwab-options/  ← BACKUP

~/.schwab/credentials.json    ← {client_id, client_secret, callback_url}
~/.schwab/tokens.json         ← {access_token, refresh_token, saved_at}
~/.databento/credentials.json ← {api_key, key_name}
~/.venvs/schwab-options/      ← venv (NOT on OneDrive; regenerable)
```

**Credentials are deliberately outside the project tree** — that way if the project ever goes to git, no secret leak risk. Also lets multiple projects share the same Schwab / DataBento credentials later.

## Auth lifetimes worth knowing
- **Schwab access token**: 30 min (auto-refreshed by `schwab_auth.get_access_token()`)
- **Schwab refresh token**: 7 days → weekly `python schwab_auth.py login` to re-auth
- **DataBento API key**: indefinite, but freely rotatable on the portal (no approval queue, unlike Schwab)

## See also
- [[00 - Overview]]
- [[03 - Math & Sign Conventions]]
- [[04 - Symbology & Gotchas]]